Skip to content

vtgate: add viper config to ignore specified query shapes from query log#20174

Open
ChaitanyaD48 wants to merge 4 commits into
vitessio:mainfrom
ChaitanyaD48:vtgate/querylog-ignore-list
Open

vtgate: add viper config to ignore specified query shapes from query log#20174
ChaitanyaD48 wants to merge 4 commits into
vitessio:mainfrom
ChaitanyaD48:vtgate/querylog-ignore-list

Conversation

@ChaitanyaD48
Copy link
Copy Markdown
Contributor

@ChaitanyaD48 ChaitanyaD48 commented May 23, 2026 •
edited
Loading

Description

Operators can now suppress specific SQL query shapes from the vtgate query log via a new opt-in --query-log-ignore-patterns flag. Empty by default.

How it works

  • New package go/vt/vtgate/querylogignore holds the flag (Viper-managed, dynamically reloadable), the parsed IgnoreSet, and the ShouldIgnore lookup.
  • Matching uses the canonical normalized form: queries and patterns are both run through sqlparser.RedactSQLQuery, then type annotations (/* INT64 */) and bind-variable names (:vtg1, :redacted1, :x) are stripped so equivalent shapes compare equal regardless of literal values. Patterns that fail to parse fall back to a case-insensitive, trimmed raw-string match.
  • Integration lives in the executor (sendQueryLog helper, called from the three queryLogger.Send sites in Execute, StreamExecute, and Prepare).

Usage

  1. Inline: --query-log-ignore-patterns='select 1 from dual,select $$,PING'
  2. From a file (@ prefix; one pattern per line; # comments and blank lines are skipped): --query-log-ignore-patterns='@/etc/vt/ignored-queries.txt'
  3. Via a Viper config file, which enables dynamic reload on config-file changes:
query_log_ignore_patterns: "select 1 from dual,select $$"

Test plan

Automated tests

  • New unit tests in go/vt/vtgate/querylogignore/querylogignore_test.go.
  • Executor-level integration test in go/vt/vtgate/executor_querylog_ignore_test.go.
  • Flag help-output regression: TestHelpOutput/vtgate, TestHelpOutput/vtcombo, and TestHelpOutput/vtgateclienttest all pass with the new flag added to go/flags/endtoend/vtgate.txt and go/flags/endtoend/vtcombo.txt.

Manual testing via examples/local

Brought up a local cluster (./101_initial_cluster.sh) with vtgate configured in following ways:

a) CLI flag, inline list
--query-log-ignore-patterns='select 1 from dual,PING'

b) CLI flag, file form
--query-log-ignore-patterns='@/tmp/ignored.txt' where /tmp/ignored.txt contained patterns one per line, plus blank lines and #-prefixed comments to confirm they're skipped.

c) Viper config file (for dynamic-reload verification)
--config-file=/tmp/vtgate.yaml
--config-file-not-found-handling=error with /tmp/vtgate.yaml containing query_log_ignore_patterns: "@/tmp/ignored.txt".

For each setup, in two terminals:

  • curl -sN http://localhost:15001/debug/querylog to stream the log.
  • mysql -h 127.0.0.1 -P 15306 -e '<query>' to send traffic.

Verified:

  • select 9999 from dual is suppressed when select 1 from dual is configured (different literal, same canonical shape).
  • PING is suppressed via raw-string fallback.
  • select $$ is suppressed (parses successfully in current sqlparser; both pattern and query land in the same canonical select $$ from dual bucket).
  • A non-matching query (select * from customer) still appears in the log stream.
  • After editing /tmp/vtgate.yaml and saving, the new patterns take effect on the next query without restarting vtgate (Viper dynamic reload via fsnotify).

Related Issue(s)

Fixes #20173

Checklist

  • "Backport to:" labels have been added if this change should be back-ported to release branches
  • If this change is to be back-ported to previous releases, a justification is included in the PR description
  • Tests were added or are not required
  • Did the new or modified tests pass consistently locally and on CI?
  • Documentation was added or is not required

@ChaitanyaD48
vtgate: add query-log ignore-list package
0c08564 
Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
@ChaitanyaD48
vtgate: route query-log emission through ignore-list
79b074f 
Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
@ChaitanyaD48
vtgate: add --query-log-ignore-patterns to help reference
c3a940f 
Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
@ChaitanyaD48
test: add test for querylogignore.go
5a3841e 
Signed-off-by: ChaitanyaD48 <chaitanya.d48@gmail.com>
Copilot AI review requested due to automatic review settings May 23, 2026 22:19
@vitess-bot vitess-bot Bot added NeedsWebsiteDocsUpdate What it says NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request labels May 23, 2026
@github-actions github-actions Bot added this to the v25.0.0 milestone May 23, 2026
@vitess-bot vitess-bot Bot added the NeedsBackportReason If backport labels have been applied to a PR, a justification is required label May 23, 2026
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot Bot commented May 23, 2026

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

Copilot AI reviewed May 23, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an opt-in vtgate query-log ignore mechanism so operators can suppress configured SQL query shapes from /debug/querylog output. It introduces a dynamically reloadable Viper flag, query-shape normalization logic, executor integration, and tests/help-output updates.

Changes:

  • Added querylogignore package for parsing inline/file-backed ignore patterns and matching normalized query shapes.
  • Routed vtgate executor query-log emission through a new sendQueryLog helper.
  • Added unit/integration coverage and updated vtgate/vtcombo flag help snapshots.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
go/vt/vtgate/querylogignore/querylogignore.go Adds flag registration, pattern loading, canonicalization, and ignore matching.
go/vt/vtgate/querylogignore/querylogignore_test.go Covers ignore-set parsing, normalized matching, raw fallback, file loading, and empty fast path.
go/vt/vtgate/executor.go Applies ignore matching before sending vtgate query-log entries.
go/vt/vtgate/executor_querylog_ignore_test.go Verifies executor-level suppression and non-matching query logging.
go/flags/endtoend/vtgate.txt Updates vtgate help output for the new flag.
go/flags/endtoend/vtcombo.txt Updates vtcombo help output for the new flag.

Comment thread go/vt/vtgate/querylogignore/querylogignore.go
GetFunc: func(v *viper.Viper) func(key string) *IgnoreSet {
return func(key string) *IgnoreSet {
newVal := v.GetString(key)
if cur, ok := v.Get(key).(*IgnoreSet); ok && cur.source == newVal {
@ChaitanyaD48 ChaitanyaD48 mentioned this pull request May 23, 2026
Closed
5 tasks
@promptless
Copy link
Copy Markdown
Contributor

promptless Bot commented May 23, 2026

Promptless prepared a documentation update related to this change.

Triggered by PR #20174

Added documentation for the new --query-log-ignore-patterns flag to the vtgate and vtcombo reference docs, the monitoring user guide (with a new "Filtering the Query Log" section), and the v25.0 changelog.

Review: vtgate: add viper config to ignore specified query shapes from query log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@harshit-gangal harshit-gangal Awaiting requested review from harshit-gangal harshit-gangal is a code owner

@systay systay Awaiting requested review from systay systay is a code owner

@frouioui frouioui Awaiting requested review from frouioui frouioui is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

Component: VTGate NeedsBackportReason If backport labels have been applied to a PR, a justification is required NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says

Projects

None yet

Milestone

v25.0.0

Development

Successfully merging this pull request may close these issues.

Feature Request: Configurable query-shape ignore-list for vtgate query log

2 participants

@ChaitanyaD48